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Abstract 

We register a random sequence constructed based on Markov processes by switching 
between them. At unobservable random moment a change in distribution of observed 
sequence takes place. Using probability maximizing approach the optimal stopping 
rule for detecting the disorder is identified. Some explicit solution for example is 
also obtained. The result is generalization of Bojdecki's model where before and 
after the change independent processes are observed. Keywords. Disorder problem, 
sequential detection, optimal stopping, Markov process, change point. 



1 Introduction 



This paper deals with a special problem belonging to the wide class of disorder 
problems. Suppose that the process X = {X n ,n € N}, N = {0, 1,2,.. .}, is observed 
sequentially. It is obtained from Markov processes by switching between them at 
random moment 9 in such a way that the process after starts from the state 
Xg^\. Our objective is to detect this moment based on observation of X. There 
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are some papers devoted to the discrete case of such disorder detection which gen- 
eralize in various directions the basic problem stated by Shiryaev in [9] (see e.g. 
Brodsky and Darkhovsky [5], Bojdecki [3], Bojdecki and Hosza [1], Yoshida |15j . 
Szajowski HUH]). 

Such model of data appears in many practical problems of the quality control (see 
Brodsky and Darkhovsky [5j, Shewhart [8] and in the collection of the papers [2]), 
traffic anomalies in networks (in papers by Dube and Mazumdar [6], Tartakovsky et 
al. [13]), epidemiology models (see Baron [1]). The aim is to recognize the moment 
of the change the probabilistic characteristics of the phenomenon. 

Typically, disorder problem is limited to the case of switching between sequences 
of independent random variables (see Bojdecki [3]). Some developments of basic 
model can be found in p3] where the optimal detection rule of switching moment 
has been obtained when the finite state-space Markov chains is disordered. Mous- 
takides [7] formulates condition which helps to reduce problem of quickest detection 
for dependent sequences before and after the change to the case of independent pro- 
cesses. Our result is generalization of results obtained by Bojdecki in [3j. It admits 
Markovian dependence structure for switched sequences (with possibly uncountable 
state-space). We obtain an optimal rule under probability maximizing criterion. 

Formulation of the problem can be found in Section [2) The main result is presented 
in Section [3l Section U] provides example of application for considered model. In 
appendix we derive useful formulas for conditional probabilities. 



2 Formulation of the problem 

Let (fi, T, P) be a probability space which supports sequence of observable random 
variables {X n } ng M generating filtration T n = a(Xo,X\, ...,X n ). The sequence takes 
values in (E, B), where E is a subset of Ji. Space (fi, J 7 , P) supports also unobserv- 
able (hence not measurable with respect to F n ) variable 9 which has geometrical 
distribution: 

P(0 = j) = p>-\ q = 1 -p G (0, 1), j = 1, 2, ... (1) 

For x G E we introduce also two homogeneous Markov processes (X®,G®,P®), 
(X^, ^,Pi) (both independent on 9), which are connected with {X n } and 9 by the 
following equation: 

X n = X® ■ I{ e>n } + X\ ■ \x\_ x =X Q e _ v 6<ny ( 2 ) 
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We have that: Q n = a(X l , X\, . . . , X*j, i G {0,1}, n G {0,1,2,...}. On (E, B) 
for x G E there are defined u-additive measures and /i l x (i = 0, 1) satisfying 
following relations: 



PI({lo : X\ G B}) =P(X{ G B\X* = x) = f fM^dy) 

Jb 

£(dy) = lj x {B). 



B 



for any B G B. 

Let us now define function S, G 



S(x J0: n) = ^2p l 1 qL n - i+1 (x 0tn ) +p n L {x 0n ), (3) 



i=l 



G(x n _ z _ ljn , a) = ai i+ i (x n _j_i jTl ) + (1 - a) (4) 
x fep'" i 9^i+i(£ n -l-i I n)+l'' + %(2»i-I-l I jl • 

where xo, sci, . . . , a: n G E n+1 , a G [0, 1], < n — I — 1 < n. Here we use the following 
notation: 



— k n — [p*k j -^fc+l ) •• •) 1 5 5 k ^ TL, 
n—m n 

Lm(x_k,n) = JJ fx r -i( X r) Y\. fx r -i( X r)i 
r=fc+l r=n— m.+l 

iifc,n = x"=fcA = ^fc x ^fc+i x ... x An, Ai e B 
where the convention that Ili=ji x « = 1 f° r h > J2 holds. 

Function 5(x n ) stands for join density of vector X_ n . For any D n = {uj : 2^o. n e 
-Bq , Bi G 23} and any x G E we have: 



P x(^0,n) = P(I2o,nl*0 = x) = / )/i(dx Q ) 

The meaning of function G(x fc n , a) will be clear in the sequel. 

Shortly speaking our model assumes that process {^ n } is obtained by switching 
at random and unknown instant 9 between two Markov processes {X®} and {X n }. 
Notice that what we assume here is that the first observation Xq after the change 
depends on the previous sample Xq_\ through the transition pdf fx g _ (Xe). During 
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on-line observation of {^n} we a i m in detection of switching time 9 in optimal way, 
according to the maximum probability criterium. For any fixed d G {0, 1, 2, ...} we 
look for the stopping time r* £ T such that 

P*(|0 - r*| < d) = sup P x (\9 - r| < d) (5) 

where S x denotes the set of all stopping times with respect to the filtration 
{•^njneN- Using parameter d we control the precision level of detection. The most 
rigorous case: d = will be studied in details. 



3 Solution of the probblem 

Let us define: 

Z n = P x (\6 - n\ < d | F n ), n = 0, 1, 2, . . . , 

V n = ess sup P x (|# — n| < d | JF n ), n = 0, 1, 2, . . . 

{r66 x , T>n} 

r = inf{n : Z n = V n } (6) 

Notice that, if Z M = 0, then Z T = P x (|# — t\ < d \ !F T ) for r G S x . Since T n C J 7 ,- 
(when n < r) we have 

K = ess sup P x (|6> - r| < d | = esssupE x (I{| e _ r |< d } | T n ) 

r>n r>n 

= esssupE x (Z r | J^n) 

r>n 

The following lemma ensures existence of the solution 

Lemma 1 The stopping time tq defined by formula ^ is the solution of problem 
PROOF. From the theorems presented in [3] it is enough to show that lim Z n = 0. 

n— »oo 

For all natural numbers n, k, where n > k we have: 

Z n = ^x(\\9-n\<d} I Fn) < E x (sup \\d-j\<d] I ^n) 

From Levy's theorem lim sup n _ >00 Z n < E x (sup J>fc I{|e_j|<^} | J 7 ^) where J 7 ^ = 
° (U^i Fn) ■ It is t ru e that: limsup J>fc) k-*ao^-{\9-j\<d} = a - s - an d by the domi- 
nated convergence theorem we get 

lim E x (supI{| e _ J |< d } | J 7 ^) = a.s. 
what ends the proof of the lemma. 
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Lemma 2 Let r be a stopping rule in the problem Then rule f = max(r, d+ 1) 
is at least as good as r. 

PROOF. For r > d + 1 rules r, f are the same. Let us consider the case when 
r < d + 1. We have f = d and given the fact that P z (^ > 1) = 1 we get: 

Pxdfl - r| < d) = P x (t -d<6<T + d) 
= Px(l < < r + d) 
<Px-(l < < 2d+ 1) 
= P x .(f -d<0<f + cZ) 
= P !B (|0-f|<d). 

In consequence we can limit the class of possible stopping rules to i.e. stopping 
times equal at least d+l. 

For further considerations let us define posterior process: 

n = o, 

Il n = P x (6<n\T n ), ra = l,2,... 

which is designed for information about distribution of disorder instant 9. Next 
lemma transforms payoff function to the more convenient form. 

Lemma 3 Let 

^^-{l-P +^g i ^ ofaw2 ) J( 1 -^ < 7 > 

where x±, Xd-\-2 S E, a G (0, 1) i/ien 

P a; (|0-n|<d)=E a; [^X^^ILO] 

PROOF. We rewrite initial criterion as the expectation 

P*(|0-n| <d)=E x \P x (\e-n\<d\F n )] 

= E X [P x (6 <n + d\F n )- P x (6 <n-d-l\F n )} 

Probabilities under expectation can be transformed to the convenient form using 
lemmata [9] and EJ Next, with the help of Lemma [TU1 (putting I = d) we can express 
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P x (9 < n + d | Tn) in terms of Tl n . Given this some straightforward calculations 
imply that: 



p x (\e -n\< d I r n) =(i- P d + q d f: L m m r ( f;- d - l ' n \ ) (i - n»)- 

V L oULn-d-l,n) J 



Lemma 4 Process {r] n } n >d+i where r] n = (X.n-d-1 m^-n) forms a random Markov 
function. 



PROOF. According to Lemma 17 pp 102-103 in [TO] it is enough to show that 
r) n +i is a function of previous stage r] n and variable X n+ \ and that conditional 
distribution of X n+ \ given T n is a function of rj n . For x\, x^s 6 E, a £ (0, 1) let 
us consider a function 



We will show that %+i = f(r]n, X n+ \). Notice that by Lemma [TOl (I = 0) we get 



_ f Xn (Xn + i)(q+pU n ) 

n+1 ~ — r^Y fn — • ^ ' 



Hence 



ip(rj n , X n+1 ) = (p(X n _ d _ 1>n , n re , X n+1 ) 

_( Y Y /l n (x n+1 )( g + P n n ) ^ 

- A n -d,n) A i+l) r>(V TT \ 

= (^.n-d,n+l'n„,+l) = ?7n+l. 

Define .F n = o"(6*,X 0n ). To see that conditional distribution of X n+ \ given T n is a 
function of r/ n , for any Borel function u : E — > K let us consider the conditional 
expectation of u{X n+ \) given T n : 
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B x (u(X n+1 ) | T n ) 

= E X (u(X n+1 )(l - n n+ i) | (u(X n+1 )U n+1 | 

= E X (u(X n+ i)I| e>n+1 } | F n ) + E x (n(X n+ i)I{5i< n+1 } | Tn) 

= E X ^E x (n(X n+ i)I{5) >n+1 } | Tn) | .T^J + E x ^^(m^+i)!!^^!} | jF n ) j J^n^ 



Ea; (j{0 >n+ i}E K («(X r j, + i) I .F n ) I J^nJ + Eje (l{0< n+ i}Ez(tt(X n+ i) I ,F n ) I T n 

I u(y)f° Xn (y)v(dy)P x (8 > n + 1 | T n ) + / u(y)fk n (yMdy)Pa>(8 <n + l \ T n ) 
je Je 

u(y)(p(l-U n )fUy) + ( q + pU n )f 1 x (y)) f i(dy) = [ u(y)G(X n , y, U n )fi(dy) 



Here we use Lemma [A. II 



Lemmata [3] and 2] are crucial for the solution of posed problem (|5|) . They show that 
initial problem can be reduced to the problem of stopping Markov random function 
r] n = (X_ n _ d _ 1 n ,n n ) with the payoff given by equation ([7]). In consequence we can 
use tools of optimal stopping theory for finding stopping time t* such that 

B x [h(X T ,_ 1 _ d ^,Il T *)]= sup E x . [h(X_ T _ 1 _ d ^ T ,Tl T )] (9) 

To solve reduced problem ([9]) for any Borel function u : K d+2 x [0, 1] — > 5? let us 
define operators: 

T u(x 14+2 ,a) = B X [u(X n _ d>n+1 ,U n+1 ) | 2k-i-d.fi = x l d+2 ,n n = a] , 
Q u (x 14+2 ,a) = max{u(x ld+2 , a),Tu(x 14+2 , a)}. 

Lemma 5 For the payoff function h(x ld+2 ,a) characterized by ^ and for se- 
quence {r fc }£i : 



ro(x ljd+1 ) =p 



d+l j ( \ 

1 - P + Q 



^p m L (x ld+1 ) 

r ( d+l 

rk(x ld+ i) = P / f Xd+1 (x d+2 ) max + 

J * { m=l 

the following formulas hold: 



L m(x ld+2 ) . 1 , . 

-i r k-i{x 2 d +2 ) } n(dx d+2 ). 



P m L {x 14+2 )' 



Q k h 1 (x 1A+2 ,a) = (l-a)m&x \l-p d + qV] ™ Xl ' d+2 . ;r k -i(x 2d+2 ) I , k > 1, 

I m=l P L ^l,d+2) J 

T Q k h 1 (x ld+2 ,a) = (1 - a)r k (x 2 d+2 ), k>0. 
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PROOF. By the definition of operator T and using Lemma [A. 5 1 (I = 0) given that 
n n.) = (x lid+2 ,a) we get 



T h(x ljd+2 ,a) — E x [h(X. n -d,n+l> Hn+l ) I 2Ln-d-l,n ~ ^.X4+2i^-r 



ft 



d+l 



A \ — ^ J -'m 

l-p + q 2^ -^fZHr : ~ ) ^ ~ nn+1 ) I ^n-d-l,n = £l,d+2, n « = « 



-p(l-a) / l-/ + g ^ 

^ \ m=l 



\P m L Q {X n _ dn+1 

L m-lfe,d +2 ) fx d+2 (%d+3) \ fx d+2 (xd+3)G(x d+24+3 , ft 



P m L (x 24+2 ) f° (x d+3 ) J G(x d+2td+3 , a) 



n(dx d+3 ) 



■ p(l - a) 
■- (1 - a)p 



1 d . L m ^i(x 2d+2 ) 

1-p +Q2^-^f-7z V 

~j_P L o{x 2id+2 ) 



x d+2 (Xd+3)fJ'( d Xd+3) 

(1 - a)r {x 2 d+2 ). 



Directly from the definition of Q results that 



Q h Ui,d+2i a ) = max {HX14+2, «)l t M^m+2, ")} 

f d+1 L m l, 

= (1 — a) max < 1 — p d + g — 



Suppose now that Lemma [5] holds for TQ k l h and Q k h for some k > 1. Then 
using similar transformation as in the case of & = we get 

TQ k h(x ljd+2 ,a) 



E, 



Q fc ^(^n-d,n+l' n n +l) I 2£. n -d-l,n ~ X.i >d+2 ,Tl n — ft 





maxj 







(i+l 



l-p d + qJ2 



Lm(x 2d+3 ) 

[P m L (x 24+3 ) 



;r k -i(x3,d + 3) A 1 - «)p/° d+2 (^+3) 



fj,(dx d+3 ) 



(1 - ft)r fc (x 2:d+2 ). 



Moreover 



Q + Vl ( ^l,d+2,«) 



max [h(x l d+2 , a); 

H \ fi d 1 -^m (^l,d+2) / s 

(1 - a) max U - p + g ^ — -; r k {x 24+2 ) 

{ m =lP L \.2Ll,d+2) 
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This completes the proof. 



The following theorem is the main result of the paper. 
Theorem 1 

(a) The solution of problem |5)] is given by: 

r* = M{n >d + l:l- P d + q J2 t^'^K > r'C^J} (10) 

m=1 P ^0{A.n-d-l,n) 

where r*(X n _ dn ) = lim^^ r k (X n _ dn ) 

(b) Value of the problem. Given Xq = x maximal probability for f5|) is equal to 



Px ( \0~T*\<d) 

= pd+1 L max I 1 ~ pd + * E XlT^y 

JE d + 1 [ m=l P ^Q^'—hd+l) 

xL (x,x ljd+1 )n(d(x,x l4+1 )). 



PROOF. Part (a). According to Lemma[2]we look for stopping time equal at least 
d + 1. From optimal stopping theory (c.f |10j ) we know that tq defined by © can 
be expressed as 

r = inf{n > d + 1 : h(X n ^ n ,U n ) > Q* h(X n ^ n ,U n )} 

where Q*h(X_ n _ l _ d n , H n ) = lim fc >oc Q k h(X_ n _ 1 _ d n , U n ). According to Lemma[5j 



■ r f ^ J i 1 i d , ^rn(JLn-d-l,n) 
tq = mi < n > d + 1 : 1 — p + q > 



m- 
d+1 



^p m L (X n _ d _ 1>n ) 



:inf^n>d + l:l-p + Q n ™ T (y - r G£n-d,J 



■ T . 



^P m L (X n _ d _ h% 

Part (b). Basing on known facts from optimal stopping theory we can write: 
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p* ( \e-r*\<d) 

= E X . (h^(X_ od+1 ,Il d+1 )) 



: E X (1 -II d+1 )max 



{d+i 
m=l 



d+l) 



-; r*(X 1>d+1 ) 



^ Ex U x (I {e>d+1} I F d+1 ) max { 1 - / + g £ ^Tr^^ v 



d+l 



E 



Lm{Xj 



0,d+l, 



-; r*(x M+1 ) 



x I { e>d+i} max <^ 1 - / + q £ - L (x 
= P.(0 > d + 1) £ ; max J 1 - p rf + 9 £ jTr^fl:^ V r *(x M+1 ) 



What ends the proof. 



4 Example 



Let us consider the case d = 0. Then, optimal rule (|10p reduces to simpler form 



^ M{ " si: SrS) sr * (xj} 



with 

r*(X n )=p[ /l(n)max{ J^M, r *( u )}d/i(u) 

Moreover suppose that the state space E = {0, 1}. Matrices of transition probabili- 
ties and conditional densities are as follow 



woo] Si 



0.1 0.9 



0.8 0.2 



[/40')]!-JSi 



0.7 0.3 



0.4 0.6 



1 1 
1 1 



; [/ i (j)] j=Q,l 



7 1/3 

1/2 3 
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For such model we find threshold r*(i), i = 0, 1 solving the system of equations 
r*(i)= ^p/°(i)max{^-,r*(i)Mi); i = 0, 1 

Treating r* as a function of parameter p we obtain: 



7 + 9p 35 + 27p 35 — 7p 

r;(o) = 1[o, pi] (p) + — WiCp) + (p) + so-iop-aep* 1 ^ 

♦ m , ^ , 30 + 28p Up 
r p (l) = l [0jH (p) + 5Q _ 36p2 l( P2 ,p 3 ](y) + 25 _ 50 _ 18p 2 1 (P3,i](p) 



where: p\ = |, j>2 = ^^|~ 7 , J>3 = ^^^p 1 -. The most interesting case takes the 
place when p > p% ~ 0, 946 because then the average disorder time is not too small. 
Obtained stopping rule r* depends on observations collected at times r* — 1 and 
r*. Thus, to make optimal rule more clear we need to analyze all possible sequences 
of (X T *_i,X T *) i.e. {0,0}, {0,1}, {1,0}, {1,1}. 
Sequence {0, 0}: 

In this case we stop if only | > 50 _^Q~^ 6p 2 ■ Solving the inequality for p, we get 
that stopping time takes the place for all p £ (^3, 1). 
Sequence {0, 1}: 

It reduces to inequality ^ > 25 _ 5 q 4 ^ 18 $ . Taking into account that p G (j»3, 1) a set 
of solutions is empty. 
Sequence {1, 0}: 

Pair {1,0} implies the stopping time if | > 5Q _^~ 7 g 6 ^ a . However there is no solution 
for p £ (p 3 ,l). 
Sequence {1, 1}: 

This sequence rises the alarm if only | > 25-50^-18?^ ' ^ turns out that the inequality 
is satisfied for any p £ (p3, 1). 

The analysis shows that we obtain very clear and simple optimal rule for case p > p%: 
stop at the first moment when two "zeros" or two "ones" occur in a row. 



A Lemmata 

Lemma 6 Let n > 0, k > then: 

p x (e <n + k\ F n ) = i- P k (i-n n ). (A.i) 

PROOF. It is enough to show that for D G T n 
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/ h 6>n+ k}dPx= / p fc (l-n n )dP x . 

JD JD 

Let us define T n = o-{F n , I{e>n})- We have: 



{6>n+k}dP x — / I{6»>n+fc}I{6»>n}^Px- — / ^{9>n+k}dPx 
D JD JDn{6>n} 



/ E x(I{0>n+fc} | -?n)dPi — / E x (I{ 9>n+k y | > n)t 

JDn{6>n} JDn{e>n} 

I i {e>n]P k dP x = [ (i - n n ) P fc dP x 

■JD JD 



Lemma 7 For n > i/te following equality holds: 



P x (0>n\T n ) = l-U n = J}r°: n . (A.2) 



PROOF. Put £> 0n = {u : X o n G A^, Aj € £}. Then: 

_ f P nL °&0,n) s(x )u(dx ) _ f P nL ^n) dr 
JA „ b{Xa,n) J Don MA ,J 

Hence, by definition of conditional expectation, we get the thesis. 



X 



Lemma 8 For Xq i+1 G E i+2 , a € [0, 1] and functions S, G given by equations 
and we /iaue: 



5(Xo )n ) — S(2Lo,n-l-l)G(2L n -l-l,m ^-n-l-l) (A. 3) 



PROOF. By (TP]) we have 
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S ( Xo,n-l-l)G(l£-n-l-l,n^n-l-x) 

= S(^ n _ l _ 1 )U n -i- 1 Li +1 (X n _ l _^ n ) + 5(X n _;_ 1 )(l - n n _j_x) 
x (^p^qLk+^X^^) +p l+1 L (X n -i-i,n)) 

\k=0 / 



I 



X J^p' fe «ifc+l(^H,„)+p l+1 io(^H,„) 

Vfc=0 / 
n-l-l I 

= P^qLn-k+ii^n) + J2p n - h - 1 qL k+1 (X J0 ^+p n L (X J 

k=l k=0 
n—l—l n 
= P fc_1 ^n-fe+l(^0,J+ Yl P k ' 1 1 L n-k+l(X 0:n )+p n L (X 0:n ) 

k=l k=n—l 
n 

= ^/-Vn-fc+lQ^n) +P n L (X , n ) = 5(X 0jn ). 
k=l 

Lemma 9 For n > I > the following equation is satisfied: 
P x (0 <n-l-l\F n ) = — — r . 

PROOF. Let L>o,n = : 2L Q ,n 6 Ao, n >^ e Then 

Px ( ^ 0n )P a: (e>n-Z-l| ; D 0n ) = / I {e>n -i-i}dP x = [ P x {6>n-l\T n )dP 
Ek=n-l p x(0 = felWggj + Px(0 > n)L (x 0; J ^ 

^7 n '-'(^0,nJ/-H«^0,nJ 



x <S'(^0,ri)/ / (^3i0,n) 

p" +1 ioKn-i-i) (E l k= p l ~ k qLk + i(x n ^ hn ) + P l+1 L (x n _ l _ hn : 



dP, 



n-l-l; 



{EM f n ,Ei=0P l ~ k Q L k+l(Kn-l-l,n)+P l+1 L (X n + hn ) 

= / (l-n n _i_x) — r dP 

•ADo.n ^V±-n-l-l,n-> lL n-l-l) 



X 
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What implies that: 



P x ( e >n-l-l\T n ) (A.4) 

n „ ,EL P i " fc ? i Hl&-i-l,J+?' !+1 io(l n -|-l,„) 

- (1 - ll„_/_lj fj v 

Simple transformations of (IA.4j) lead to the thesis. 
Lemma 10 For n > I > recursive equation holds: 

n n ^i^ 1 Li +1 (2L n _ l _i n ) + (i -u n ^ l _ 1 )qY,i =0 p l ~ k Lk+i(2L n ~i~i,n) 
Un = Thy n S ^ A - 5 ^ 

PROOF. With the aid of (CO]) we get: 

l-n n P n L (X „) s r (x 0in _,_ 1 ) ^+ 1 L (x n _ J _ lin ) 



Hence 



n _ C(^n-Z-l,n! n n-/-l) ~ ' ±L (2Lo,n-l-l ) (j- ~ H-n-l-1 ) 

n n _;_iL Z+ i(X n „ ; _ l n ) + (1 - n n _ i _i)(?X]fc=0^~ fcL fc+l(^n-«-l,n) 
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